Repository navigation
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
✅ API Diff Results -
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pkcll
left a comment
There was a problem hiding this comment.
Thanks — this is a clean, additive change: default-off, reuses the batch emitter rather than forking it, and the in-process gRPC server tests (method, routing header, auth, resource attrs, NOP header, caller extensions) are a nice pattern. The cross-domain source guard is a good touch.
I ran the package locally at c2f5a8a (go test ./pkg/beholder/ -race: pass, 81.8% coverage; new tests ×10 with -race: no flakes) plus a few lifecycle/outage probes. Two things I'd like addressed, the rest is optional.
1. Domain emitters close their gRPC connection twice → Client.Close() always returns errors (blocking)
ChipIngressBatchEmitterService.stop() calls e.batchClient.Stop() and then e.ownedClient.Close(), but batch.Client.Stop() already closes the chipingress client (pkg/chipingress/batch/client.go, end of Stop). With the flag on, a clean start → Close() returns three extra errors:
rpc error: code = Canceled desc = grpc: the client connection is closing (×3, one per domain)
It's hidden in the tests because every Close() result is discarded (t.Cleanup(func() { _ = client.Close() }), _ = client.Close() in the drain test).
Suggestion: drop the ownedClient.Close() from stop() (Stop owns the connection once started) but keep closing it on the construction-failure paths in newDomainEmitters / NewGRPCClient, and assert in a test that a start → close with domain emitters returns no domain-related error.
2. During an outage, every dropped event logs at Error (before enabling the flag on a fleet)
queue() logs failed to emit to chip ingress inside the per-message callback for every non-PublishError failure, and failed to queue message for chip ingress for every ErrMessageBufferFull. When one domain's chip ingress (or its LB rule) is unavailable — or the server doesn't have PublishBatchNoAck yet (Unimplemented, no fallback) — each node emits one Error line per telemetry event for that domain. This is pre-existing behaviour moved into queue(), but domain emitters are going to carry node telemetry volume, so the blast radius is new.
Could we rate-limit these (e.g. first failure + periodic summary per error_code), or log per batch and rely on chip_ingress.events_dropped{error_code} for per-event accounting, as the code already does for PublishError? Fine to do as a fast follow if you prefer, as long as it lands before the flag is turned on anywhere.
Tests worth adding
- Outage/failure injection for domain emitters: server down, hung server +
Close(callbacks fire exactly once), buffer full,Unimplemented. - Concurrent
EmitEvent+Client.Close()under-race. - Construction-failure branches (
newDomainEmittersis at 76%:NewClienterror, batch-service ctor error, both cleanup defers). Emit(attribute path) happy path on a domain emitter — only the reject case is covered.
Nits (optional)
var domains = []Domain{...}is a mutable package var; a func or a fixed-size array avoids accidental mutation.append(append([]chipingress.Opt{}, opts...), ...)→slices.Concat(...).svc.ownedClient/svc.domainare set after construction; passing them through the constructor keeps it in one place.checkDomainis case-sensitive and requires the.separator; worth a line in theDomainEmitterdoc.- Mention in the
ChipIngressDomainEmittersEnableddoc that domain emitters are always batch emitters (independent ofChipIngressBatchEmitterEnabled), and the per-node cost: one buffer ofChipIngressBufferSizeper domain; a connection only for domains actually used (dialing is lazy).
Questions
- What does the LB do with an unknown or missing
chainlink-domainheader (default backend vs reject, which gRPC status)? - Is a chip-ingress server version with
PublishBatchNoAckalready deployed in stage/prod? - Will consumers (OTI) need a
beholdertestfake forDomainEmitter?
Out of scope here (I'll track separately)
pkg/chipingress/batch:splitMessagesByRequestSizerecomputesproto.Sizeover the whole candidate batch for every appended message — O(n²), and ~90% of allocations on the send path in a memprofile. Not introduced by this PR, but this PR adds three more emitters on that path.- Client shutdown order: OTel providers are shut down before the batch/domain emitters drain, so drain-time metrics/spans are lost.
Happy to approve once (1) is fixed and there's an outage test.
|
Thanks @pkcll — great review. Fixes for (1), (2) and the tests are coming in follow-up commits. Answers to your questions: 1. Unknown/missing 2. 3. |
batch.Client.Stop already closes the chip ingress client, so stop() closing it again made every Client.Close return one Canceled error per domain. The connection is now only closed directly when construction fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers server down, hung server + Close (each callback fires once), buffer full, a server without PublishBatchNoAck, concurrent EmitEvent and Close, the Emit attribute path, and construction failures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
During an outage the batch emitter logged one Error line per dropped event. It now logs the first failure per error code, then at most one per minute with a suppressed count. chip_ingress.events_dropped still counts every event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Domain list as a func so it cannot be mutated, slices.Concat for the domain emitter options, and docs for the source check (case-sensitive, "." separator) and for ChipIngressDomainEmittersEnabled (always batch emitters, per-node cost). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks again @pkcll — all addressed: 1. Double close — 0d120a6. 2. Per-event error logs — 5770d39. First failure per error code is logged, then at most one per minute with a Tests — 66af14e: server down, hung server + Nits — b3de7aa: Thanks for tracking the |
- Report queueing failures to the callback after the service lock is released, so a callback that re-enters the emitter cannot deadlock with Close (also fixes EmitWithCallback). - Require ChipIngressLogger when domain emitters are enabled. - ForDomain errors wrap ErrDomainEmittersDisabled or ErrUnknownDomain. - Reject ChipIngressDomainEmittersEnabled in the HTTP client. - Close the chip ingress client when NewGRPCClient fails after creating it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The invalid-event test now isolates CloudEvent validation (it passed on the domain check before). New coverage: a cancelled context, domain emitters with the batch emitter disabled, the per-domain client name, the logged event source, and rate-limited buffer-full logs. The concurrency test no longer calls require off the test goroutine and checks that emits overlap Close. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spell out what callers need: source and type must be low-cardinality, EmitEvent sends the event as given (seqnum is overwritten, nothing is stamped, the event must not be modified afterwards), when callbacks fire and what a nil no-ack result means, that the source check is a client-side guard, and that the Client must be started before Close to release connections. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rsEnabled Chip ingress is a background service users should not need to know about. Existing ChipIngress* settings are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: 9b4d2d3 | Previous: f49aea8 | Ratio |
|---|---|---|---|
BenchmarkKeystore_Sign/nop/in-process |
836.5 ns/op |
225.1 ns/op |
3.72 |
BenchmarkKeystore_Sign/hex/in-process |
789.1 ns/op |
287.9 ns/op |
2.74 |
This comment was automatically generated by workflow using github-action-benchmark.
| if batchSvc != nil { | ||
| subs = append(subs, batchSvc) | ||
| } | ||
| for _, domain := range allDomains() { |
There was a problem hiding this comment.
[P2] Drain domain emitters before shutting down OTel providers
The service engine runs the parent Close hook before closing sub-services. The parent hook calls c.close(), which shuts down the OTel meter provider; the domain batch emitters are drained afterwards. If a queued event completes during that drain, its callback records the sent/error/drop metric after the provider has shut down, so that outcome may not be exported.
Could teardown drain the domain emitters before shutting down the providers?
There was a problem hiding this comment.
Good catch, thanks — fixed for the domain emitters in 9b4d2d3. Client.close now drains them (concurrently) before the providers shut down, and a close-once wrapper makes the engine's later sub-service Close a no-op. TestForDomain_DrainsBeforeProvidersShutDown checks that the emitter has already drained and stopped when the provider shutdown runs.
I left the default batch emitter's ordering as it is to keep this PR scoped to the new code: it has drained after provider shutdown since 4a2d236, and Pavel flagged it as a separate follow-up. Happy to apply the same change there in its own PR.
The service engine runs the Client's Close hook, which shuts the OTel providers down, before closing sub-services, so outcomes recorded while the domain emitters drained were not exported. The hook now drains the domain emitters first; a close-once wrapper makes the engine's later sub-service Close a no-op. The default batch emitter is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@pkcll @thomaska ready for another look — the PR is at 9b4d2d3. Since the last round:
|
There was a problem hiding this comment.
🟡 Changes recommended
Domain validation and construction failures leave both initialized log providers running, leaking batch processors and the shared exporter connection without a returned Client to close them.
1 open issue
What changed in this PR
Adds opt-in, per-domain chip ingress emitters to Beholder, supporting telemetry routing to dedicated product-domain deployments.
Changes:
- Adds
ForDomainandEmitEventwith domain validation, dedicated buffers and connections, and no-ack publishing. - Integrates emitter lifecycle management, rate-limits failure logs, and moves queue-failure callbacks outside the service lock.
- Updates the chip ingress dependency and adds routing, shutdown, and failure-path tests.
| File | Description |
|---|---|
pkg/beholder/testdata/config-example.json |
Shows domain emitters disabled by default. |
pkg/beholder/httpclient.go |
Rejects domain emitters for HTTP clients. |
pkg/beholder/error_log_limiter.go |
Adds per-error log rate limiting. |
pkg/beholder/error_log_limiter_test.go |
Tests suppression counts and logging intervals. |
pkg/beholder/domain_emitter.go |
Defines domain APIs and constructs dedicated emitters. |
pkg/beholder/domain_emitter_test.go |
Tests routing, validation, and lifecycle behavior. |
pkg/beholder/domain_emitter_failure_test.go |
Tests outages, buffering, concurrency, and cleanup. |
pkg/beholder/config.go |
Adds the domain-emitter configuration flag. |
pkg/beholder/client.go |
Integrates domain construction, lookup storage, and shutdown. |
pkg/beholder/batch_emitter_service.go |
Adds CloudEvent emission, domain guards, and callback handling. |
pkg/beholder/batch_emitter_service_reentrant_test.go |
Tests callback re-entry during shutdown. |
go.sum |
Updates chip ingress dependency checksums. |
go.mod |
Bumps chip ingress for no-ack batch support. |
Review effort: Balanced
| if lggr == nil { | ||
| lggr = pkglogger.Nop() | ||
| } | ||
| chipIngressEnabled := cfg.ChipIngressEmitterEnabled || cfg.ChipIngressEmitterGRPCEndpoint != "" |
| func (*domainEmitter) Close() error { return nil } | ||
|
|
||
| // closeOnce wraps a service so that only its first Close does anything; later calls return nil. | ||
| type closeOnce struct { |
There was a problem hiding this comment.
Historically we have tried to avoid these because they suppress problems that should be resolved, like unclear ownership. What was the purpose in this case? Is there a logical race to close from multiple places?
There was a problem hiding this comment.
Not a race exactly: the double close request is deterministic, and it exists because of the ordering issue discussed earlier (#discussion_r4456767601, fixed in 9b4d2d3).
The engine runs the parent Close hook (which shuts down the OTel providers) before sub-service teardown. The domain emitters must drain before the providers die, or their drain-time metrics are lost, but they also need to stay sub-services for start rollback and health. So the same service is closed twice on purpose: Client.close drains it early, and the engine's later close is then a no-op. That's what closeOnce encodes.
Is that a bad idea? Should we do it differently?
There was a problem hiding this comment.
Being a sub-service usually means that the parent service owns the lifecycle. Do both operations need to strictly satisfy the Closer interface? Or could we call it something like stop or drain for the other case?
If we are stuck with Close in both cases, and if the engine always calls it first, how about wrapping only in the subservice case, and using something that suppresses close entirely? e.g. type noopCloser struct
There was a problem hiding this comment.
added in #e141f09 if you could have a look please
In the rollback scenario though, the subservice close is never run, which in this case is ok as the domain emitters cannot fail start up, but in principle they could leak resources. This is ok right?
This reverts commit b0490ab.

INFOPLAT-19544
What
beholder.Domainconstants (data-feeds,data-streams,ccip) andConfig.ChipDomainEmittersEnabled(off by default)NewGRPCClientbuilds one chip ingress emitter per domain:chainlink-domain: <domain>headerPublishBatchNoAck; NOP lookup on; client namebeholder_<domain>ChipIngressLogger; rejected by the HTTP clientClient.ForDomain(Domain) (DomainEmitter, error): lookup only; errors wrapErrDomainEmittersDisabledorErrUnknownDomainDomainEmitter=Emitter+EmitEvent(ctx, ce.Event, callback): queues a fully built CloudEvent as given (also added toChipIngressBatchEmitterService); the contract (cardinality, callbacks, no-ack semantics) is documented on the interfacedata-feeds,data-feeds.*), so an event can't take another domain's routesuppressed_since_last_logcount;chip_ingress.events_droppedstill counts every eventClose(also fixesEmitWithCallback)NewGRPCClientcloses its chip ingress client if construction fails after creating itpkg/chipingressto8ed7b172(PublishBatchNoAck client side #2478) forbatch.WithNoAckWhy
chainlink-domain(proven in the INFOPLAT-19544 spike).Notes
chipingress.Clientimplementers. The pin bump addsPublishBatchNoAckto the interface (viapb.ChipIngressClient). api-diff covers only the root module, so it doesn't flag this. Hand-written fakes must add the method when bumping, e.g. chainlink'snoopChipIngressPublisherincore/services/synchronization/chip_ingress_batch_worker_test.go.PublishBatchwith no routing header):Stoppublishes every queued message instead of dropping all but the current batch (chipingress/batch: publish queued messages on Stop instead of discarding them #2461), and a nilPublishBatchresponse fails events withRESULTS_MISMATCHinstead of panicking (chipingress/batch: treat nil PublishBatch response as RESULTS_MISMATCH #2460)🤖 Generated with Claude Code